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■  Background 

■  Value  Added  Analysis  process  using  Vector-In-Commander  (VIC) 
to  assess  trades 

■  Standard  practice  of  employing  fractional  factorial  experimental 
designs  plus  Analysis  of  Variance  (ANOVA) 

■  Non-monotonic  relationships  between  parameters  and  VIC 
outcomes 

■  Issue 

■  Statistical  assumptions  underlying  ANOVA  in  this  context 

■  Questions 

■  Are  these  assumptions  valid? 

-  How  to  test  them  and  assess  impact? 

■  Analysis  methodology 

■  Graphical  evidence 

B  Independent  replications  of  a  small  study  whose  parameters  are 
known  to  have  an  expected  value  of  zero 

B  Finding 

B  Evidence  that  the  ANOVA  assumptions  are  not  met 

B  P-values  from  ANOVA  are  understated 
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The  Value  Added  Analysis  Process 


Supporting  the  Center  for 
Army  Analysis  (CAA) 

■  Uses  the  Vector-in- 
Commander  (VIC)  Corps- 
level  combat  simulation 
model 

■  Objective: 

■  Estimate  the  incremental 
contribution  of  system 
trades  to  combat 
effectiveness 

■  Perform  a  cost-benefit 
analysis  to  determine  the 
actual  'value-added’  of 
the  systems  of  interest. 


Value  Added  Analysis 
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Figure  1.  the  VAA  Process 


Previous  methodology  was  a  typical  Design  of  Experiments  (DOE) 
approach 

Now  a  perturbation  methodology  induces  stochastic  behavior  in  VIC 
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Non-Monotonicity  in  Deterministic  Combat 
Models 


A.K.A.  Sensitivity  to  initial 
conditions  and  parameter 
values 

■  Extensively  noted  in  toy 
models  of  combat 

■  The  RAND  model 
(Dewar,  et  al,  1991) 

■  Also  noted  in  VIC 

■  Saeger  &  Hinch 
(2001) 

-  Geoff  Hawkins  (1984) 
with  VECTOR-2 


Kills  of  Selected  Equipment  by  a  Blue  System  of  Interest 


Impedes  analysis  of 
equipment  trades 
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Parametric  Sensitivity 


The  direct  fire  system 
(DF  Sys)  fraction  of 
time  firing  (FTF)  is 
multiplied  by  a 
number  randomly 
selected  from  the 
interval  (0.95, 1.05) 

Blue  kills  vary  non- 
monotonically 

■  and  significantly 

If  the  analyst  had 
chosen  a  very  slightly 
different  number,  the 
MOE  would  be 
different 


Kills  by  Blue  of  High-Value  Equipment 
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Another  Example 


The  rotary  wing  system 
(RW  Sys)  “kill  rate”  (KR) 
is  multiplied  by  a 
number  randomly 
selected  from  the 
interval  (0.95, 1.05) 
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Over  this  small  interval, 
the  true  relationship  is 
relatively  weak 

■  Non-systematic 
variation  is  much 
more  important 
than  the  systematic 
relationship 
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Slightly  Varying  Artillery  Lethal  Area 


Letha 1 _Area_M0E 


Varying  the 
artillery 
munition 
lethal  area 
within  ±5% 
has  non¬ 
monotonic 
effects  on 
the  MOE 

Notice  the 
flat  spots 
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The  Amount  of  Sensitivity  is  Not  the  Same  for 
all  Parameters 


Varying  speed 
during  scoot 

±5% 

■  Only  four 
clusters  appear 
in  the 

multivariate  data 


■  When  viewed 
in  total  or  by 
victim 
equipment 
group 


Kills  by  Blue  of  High-Value  Equipment 
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Statistical  Assumptions  in  the  Design  of 
Exper  merits 


■  A  typical  DOE  is  to  run  a 
number  of  combinations 
of  experimental  settings 

■  And  then  analyze  the 
MOEs  using  analysis  of 
variance 


■  Different  parameter 
settings  are 
superimposed  for  each 
run 


ANOVA  makes  the 
assumption  that  error 
terms  are  mean-zero  and 
statistically  independent 
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Effect  of  Superimposing  Parameter  Settings 


A  perturbation  to  the 
rotary  wing  system  kill  rate 
(KR)  is  superimposed  on 
that  of  direct  fire  system 
fraction  of  time  firing  (FTF) 

1.  Perturb  FTF 


2.  Perturb  FTF  and  KR 

If  the  first  run  had  a  high 
MOE,  the  second  one  was 
also  likely  to  be  high 

This  is  a  “memory 
effect” 


The  artifacts  from  the 
first  perturbation  are 
“remembered”  even 
though  another 
perturbation  is 
applied. 


Y 

Kills  by  Blue 
after  further 
adding  a 
perturbation 
of  the  rotary 
wing  system 
kill  rate 


Kills  by  Blue  after  perturbing  the 
direct  fire  fraction  of  time  firing 
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The  “Memory  Effect”  As  Correlation 


■  The  variability  of 
X  and  Y  are 
essentially  noise 
in  the  narrow 
parameter 
ranges 
employed. 

■  But  the  noise  is 
“correlated” 
when  a 

perturbation  in 
one  parameter  is 
added  to  the 
perturbation  of 
another 


LA  FTF  MOE 
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The  Relationship  Between  DOEs  and  ANOVA 
Assumptions 


An  example 
DOE: 


Run 
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When  slightly  different  performance  parameter  values  are 
chosen  (as  by  different  analysts  or  rounding),  we  have: 


Try  #1 

Try  #2 

Try  #3 

Run(i) 

FTF  Value 

KR  Value 

FTF  Value 

KR  Value 

FTF  Value 

KR  Value 
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Ultimately  the  model  being  estimated  is 

MOE ■  =  cc  +  f\  A-,  +  P 2  .v2  +  £j 


But  s3  is  correlated  with  e,  and  e2  due  to  the  memory  effect. 

This  is  a  deviation  from  the  ANOVA  assumptions 
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A  Model  Study  That  Should  Have  No  Findings 


A  small  study  with  an  expectation 
of  zero  findings  is  independently 
performed  64  times 

How  often  is  “statistical 
significance”  achieved? 

The  true  values  of  the  parameters 
being  estimated  are  zero 

■  The  deviations  from  the  base 
case  are  small  (±5%)  and  are 
centered  on  the  base  case 
values 

■  The  parameter  settings  are 
independently  varied  (as  if  a 
different  analyst  were  assigned 
each  time) 
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More  on  the  Model  Study 


■  64  independent  repetitions  of  a  3-factor  factorial  design 

■  Factors 

■  Fraction  of  time  firing  for  a  direct  fire  weapon 
system 

■  “Kill  rate”  for  a  rotary  wing  system,  and 

■  An  artillery  munition  lethal  area 

■  MOE  is  kills  by  Blue’s  high-value  equipment  of  Red’s 
high-value  equipment 

■  Analyzed  using  a  3-factor  main  effects  unbalanced 
ANOVA  (least  squares  regression)  model 

■  One  observation  is  missing  in  a  number  of 
repetitions 

■  More  than  one  error  d.f.  for  robustness  of  findings 

■  ANOVA  results  are  analyzed  statistically 
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Results  For  the  ANOVA  Overall  F  Statistic 


Dependent 

Variable 

Average  P- 
Value 

P-Value  of  F 
(mean=0.5) 

Number 

Statistically 

Significant 

t  statistic  (versus 
5%  level  of 
significance) 

Blue  Kills  Red 

Important 

Equipment 

0.28 

2.31  E-09 

9 

2.069* 

Red  Kills  Blue 

Important 

Equipment 

0.27 

1.00E-09 

15 

3.454** 

■  The  chance  of  Type  I  error  exceeds  the  alpha  of  5% 

■  P-Values  from  ANOVA  will  not  actually  have  the 
nominal  statistical  significance  level 

■  Similar  results  were  found  for  individual  ANOVA 
model  parameters 
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Conclusions  and  Way  Forward 


■  P-values  from  ANOVA  applied  to  DOE  data 
from  deterministic  combat  simulations  may 
not  mean  what  one  expects 

■  Due  to  the  “memory  effect”  seen  in  VIC 

■  The  way  forward  is  to  employ  perturbation 
to  randomize  the  error  terms  in  the  ANOVA 
model 

■  See  Bulanow,  et  al  2004  MORSS 
presentation. 

■  This  erases  the  “memory  effect”  and 
assures  error  term  independence  for  all 
practical  purposes. 
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